Skip to content

Add private-repo pre-flight, env-sourced gateway key, and the onboard-benchmark skill - #193

Merged
aarora79 merged 5 commits into
mainfrom
feat/workshop-private-repo-support
Sep 29, 2026
Merged

aarora79 merged 5 commits into
mainfrom
feat/workshop-private-repo-support

Conversation

@aarora79

@aarora79 aarora79 commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Why

Benchmarking a team's own repositories against a corporate model gateway hits two gaps that only show up off the published paths.

Private repositories already clone correctly: _clone_repo and the judge's clone_repo_at_ref both shell out to plain git clone with the inherited environment, so an ambient credential helper or an insteadOf rewrite works with no code change. What was missing is a check. The harness clones per task, so a bad token or a mistyped ref failed once per task, hours into a batch, as an opaque git clone failed.

The endpoint API key could also only be a literal in config/runner.yaml. That is right for a self-hosted vLLM server, which ignores the value, and wrong for a gateway token.

What changed

Repository reachability pre-flight. preflight_check.py --check-repos resolves every task's repository and pinned ref and confirms each answers, using git ls-remote so it transfers no objects and costs seconds. It separates a missing credential, a rejected token, an untrusted internal CA, an unresolvable host and a tag that does not exist, and names the fix for each. GIT_TERMINAL_PROMPT=0 means an unattended check fails rather than hanging on a password prompt. run-e2e-benchmark.sh runs it before any task starts.

Measured against the real dataset it is 0.2s for one repo, and it correctly diagnoses all three failure modes:

UNREACHABLE: .../mcp-gateway-registry @ 99.99.99 -- the repository is reachable but has
  no branch or tag named '99.99.99'. Check the ref, and remember a task should pin the
  release BEFORE its fix so the defect is present in the tree.
UNREACHABLE: .../no-such-repo @ main -- the repository path does not exist on the server,
  or the credential cannot see it.
UNREACHABLE: https://ghes.invalid.internal/org/repo @ main -- the host did not answer.
  Check the URL, DNS, the security group and any proxy between this machine and the server.

api_key_env. Names an environment variable holding the endpoint key. It wins over api_key and fails at config load when the variable is unset, rather than hours later inside the agent with an authentication error that never mentions it. Wired through all three layers per the AGENTS.md config checklist: the Pydantic field, runner.example.yaml, and a --api-key-env flag on both run-swe-headless.py and runner_config.py.

dataset/workshop-template.yaml. An annotated two-task starting point, built around pinning the release before a fix so the defect is actually present in the cloned tree. Validates as-is.

Docs. docs/benchmark-your-own-repo.md gains a section on private repositories and GitHub Enterprise Server: credential helpers, insteadOf token rewriting, deploy keys, the GIT_SSL_CAINFO and NODE_EXTRA_CA_CERTS pair (two variables, two separate failures, one root cause), and the pin-the-release-before-the-fix rule promoted out of a tip.

Security findings from the gate, and their fixes

The security-check gate flagged two issues created by letting a real token flow down a path that previously only ever carried the local placeholder. Both are fixed in this PR.

  1. The key was written to world-readable files. _write_omp_config and _write_pi_models_json emit the endpoint apiKey into a per-run models.yml / models.json via write_text, so umask left them 0644. Both are now chmod 0o600 (pattern Gate the benchmark on a >=200K context window #7: a machine-written credential belongs in an owner-only file).
  2. --dry-run printed the key. It prints the full agent command, and on the claude path that command carries --settings inline JSON containing ANTHROPIC_API_KEY and apiKeyHelper: echo <token>. Harmless at local, a credential leak into any redirected dry-run output at a real token. _redact_api_key now masks it, and deliberately leaves the local placeholder alone so paths like ~/.local/bin are not mangled.

One residual risk, documented rather than fixed. On the claude path the settings JSON is a command-line argument, so a real key is visible in the process table to any other local user. Claude Code requires a token source even against an endpoint that ignores it, and restructuring how settings are passed would touch the hot path every published result depends on. The docs now state this per-agent (omp and pi keep the key in a 0600 file and are unaffected) and point at settings_file as the mitigation. Worth a follow-up issue if anyone runs the harness on a shared box.

Verification

  • 453 tests pass, up from 425: 23 new for the pre-flight, 6 for api_key_env, 5 for the two security fixes.
  • ruff check, ruff format --check, and bandit -r scripts/ clean (0 issues at every severity).
  • mypy clean on dataset_loader.py, runner_config.py and preflight_check.py. The last was not previously in the CI mypy target and had two pre-existing errors from a -> "callable" annotation directly above the new code; replaced with Callable[[str], str] so the file is clean.
  • Exercised for real, not just unit-tested: the pre-flight against the live mcp-gateway-registry repo and against all three failure modes, and a --dry-run with a token in the environment confirming zero occurrences of the secret in the output.
  • bash -n on the modified orchestrator. The new shell step mirrors the existing preflight_check.py call two lines below it; it was not exercised end to end because this box has no gateway to run a benchmark against.

Not in scope

Nothing outstanding. An earlier version of this PR said codex was the judge and not a harness; main added it as a fifth agent (#162, #184) while this branch was open, and the merge below reconciles that.


Second commit: the onboard-benchmark skill, and drift in benchmark

Why a new skill

Nothing guided someone from "we have models and repositories" to a first scored run on their own infrastructure. setup-machine installs dependencies; benchmark runs a model through a setup that already works. The gap between them is the whole of onboarding, and it is where a first attempt actually fails.

onboard-benchmark covers it in ten steps, each ending in a proof rather than a filled-in config file:

  1. Where will this run, and does that machine reach both sides? Every task makes two calls that must both succeed from the same host: a clone from the code host and a model call to the gateway. Those usually sit on different networks, and a box reaching only one cannot run a single task. Verified from that machine, not from a diagram. A subnet or peering change is named as a blocker with days of lead time, not something to work around, and the fallback is a public repository for the first pass.
  2. Read the gateway's API shape and pick the harness that can speak it (OpenAI-compatible means omp or pi; Anthropic-compatible means claude), then settle whether its auth is a static key at all.
    3-5. Wire the agent, the judge and repository access, each proven with a real call. A successful curl does not prove the CLI works, and an unconfigured codex ignores ambient credentials entirely.
    6-10. Author the dataset, run one smoke task, settle the cost basis, scale to a batch, package the frontier as a swe-router skill.

It carries the honest timeline from measured runs: 110 to 449 minutes per 21-task model, so a first session produces one scored task and a working pipeline.

Drift corrected in benchmark

Checking whether that skill needed my pre-flight change turned up more than expected. Anyone following it today would be told the headline harness does not exist and pointed at folders that are not there.

Issue It said Reality
Agents claude and pi only Orchestrator accepts claude|pi|omp|kiro. omp appeared zero times, and it produced the published frontier
pi on Bedrock "has no native Amazon Bedrock mode, so it cannot run --provider bedrock" Wrong. PI_PROVIDER_BEDROCK = "amazon-bedrock" plus two committed scripts that do exactly that
Skill selection --skill undocumented; text described /swe2 throughout swe3 is the default
Artifact paths swe-benchmark-data/{model-slug}/<repo>/<task>/ Missing two levels. Wrong in 7 places, including the judge and summarizer commands, which as written point at a tree the harness never writes to
Default dataset the 5-task v1 set v2, 21 tasks, is what the published results come from

Also added there: the repository-reachability pre-flight and api_key_env from the first commit, a note that context_window is required behind a custom base URL, and a pointer to onboard-benchmark for a first run.

Security note on the new prose

Reviewing my own guidance caught one thing worth fixing: the credential instructions recommended credential.helper store without saying it writes the token in plaintext to ~/.git-credentials. Both the skill and the docs now say so, offer the environment-only insteadOf form and deploy keys as alternatives, and state explicitly that GIT_SSL_NO_VERIFY is never the fix for a TLS error (trust the CA instead).

Registration and verification

Registered in AGENTS.md, README.md, docs/repository-structure.md and docs/benchmark-your-own-repo.md. Every relative link in both skill files verified to resolve.

Docs and skills only in this commit, no Python or shell touched. Verified locally: all 7 pre-commit hooks pass on every file changed in the branch, ruff check, ruff format --check and mypy clean, 453 tests pass.

CI note: GitHub Actions ran green on the first commit (all 5 checks) but has not created runs for the second one, ~20 minutes after the push, and a close/reopen did not trigger them either. The repo has no path filters and no other activity to compare against, so this looks like an Actions-side delay rather than anything in the diff. Re-run or push a nudge commit if the checks are still absent at review time.


Merged main (13 commits), and what that needed

main added codex as a fifth coding agent (#162, #184) while this branch was open. That is the change that mattered; the rest merged cleanly.

Textual conflicts, all in .claude/skills/benchmark/SKILL.md:

  • The agent input. Kept this branch's table, added a codex row, and kept main's Responses-API tool-parser warning, which is hard-won and still true. Deliberately did not keep main's claim that pi cannot run --provider bedrock: pi has a PI_PROVIDER_BEDROCK constant and two committed scripts that do exactly that, and correcting it is one of the drift fixes this PR makes.
  • The CLI pre-flight. Kept this branch's structure, added codex exec --json to the agent binaries.
  • The orchestrator command line. Took both additions, codex in the agent list and --skill.

One semantic conflict git could not see. Main's _build_codex_env sets OPENAI_API_KEY from config.api_key directly, which bypasses this branch's api_key_env and would send the local placeholder to a gateway that needs a real credential. Routed through resolved_api_key(), verified live: with api_key_env: WS_TOKEN set, _build_codex_env now returns the token from the environment. All six call sites that send the key honour it.

The onboard-benchmark skill said codex was only the judge. Its harness-selection table now has a Responses-API row for codex, and both that step and the judge step note that --agent codex puts one tool on both sides of the run, so the judge model must differ from the model under test.

One test failure during the merge was an artifact, not a defect. test_committed_models_json_matches_the_generator failed while the merge was staged but uncommitted: measured_on comes from git log -1 --format=%cs on the frontier JSON, so git log still saw only this branch's history (2026-09-09) while the working-tree file was main's (2026-09-10). Committing the merge made main's commit reachable and the test passes. Confirmed the test passes on origin/main alone in a clean worktree first, so it was never a pre-existing failure.

Post-merge verification: 498 tests pass, ruff check, ruff format --check, mypy and bandit clean (0 findings at every severity), bash -n on the orchestrator. All 5 CI checks green on the merge commit.

Benchmarking a team's own repositories against a corporate model gateway hit two
gaps. Private repositories already clone fine with ambient git credentials, but
nothing checked them up front, so a bad token or a mistyped ref failed once per
task, hours into a batch, as an opaque "git clone failed". And the endpoint API
key could only be a literal in config/runner.yaml, which is the wrong place for a
real gateway token.

- preflight_check.py --check-repos resolves every task's repository and pinned
  ref and confirms each answers, using git ls-remote so it transfers no objects
  and costs seconds. It separates a missing credential, a rejected token, an
  untrusted internal CA, an unresolvable host and a tag that does not exist, and
  names the fix for each. Terminal prompting is disabled so an unattended check
  fails rather than hanging on a password prompt. run-e2e-benchmark.sh runs it
  before any task starts.
- api_key_env names an environment variable holding the endpoint key. It wins
  over api_key and fails at config load when the variable is unset, rather than
  hours later inside the agent with an authentication error that never mentions
  it. Wired through the config model, the example YAML and both CLIs.
- The per-run agent config files that carry the key (omp models.yml, pi
  models.json) are now written 0600 rather than umask-default, and --dry-run
  redacts the key it would otherwise print inside the claude settings JSON.
- dataset/workshop-template.yaml is an annotated two-task starting point, built
  around pinning the release before a fix so the defect is present in the tree.
- docs/benchmark-your-own-repo.md gains a section on private repositories and
  GitHub Enterprise Server: credential helpers, insteadOf token rewriting,
  deploy keys, the GIT_SSL_CAINFO and NODE_EXTRA_CA_CERTS pair, and where the
  key ends up per agent.

Also replaces the pre-existing `-> "callable"` annotation in preflight_check.py
with Callable[[str], str], so that file is now mypy-clean.

453 tests pass; ruff, mypy and bandit are clean.
@aarora79 aarora79 added documentation Improvements or additions to documentation enhancement New feature or request labels Sep 29, 2026
Nothing guided someone from "we have models and repositories" to a first scored
run on their own infrastructure. `setup-machine` installs dependencies and
`benchmark` runs a model through a setup that already works; the gap between them
is the whole of onboarding, and it is where a first attempt actually fails.

New `onboard-benchmark` skill, in ten steps that each end in a proof:

- Step 1 asks the question with the longest lead time, which is where the
  benchmark will run. Every task makes two calls that must both succeed from the
  same host: a clone from the code host and a model call to the gateway. Those
  usually sit on different networks, and a box reaching only one cannot run a
  single task. It must be verified from that machine, not from a diagram, and a
  subnet or peering change is not something to work around.
- Step 2 reads the gateway's API shape and picks the harness that can speak it
  (OpenAI-compatible means omp or pi; Anthropic-compatible means claude), then
  settles whether its auth is a static key at all.
- Steps 3 to 5 wire the agent, the judge and repository access, each proven with
  a real call, because a config file that looks right is not evidence.
- Steps 6 to 10 author the dataset, run one smoke task, settle the cost basis,
  scale to a batch, and package the frontier as a swe-router skill.

It also corrects the benchmark skill, which had drifted well past this change:

- Agents: it documented `claude` and `pi` only. The orchestrator accepts
  `claude|pi|omp|kiro`, and omp, which produced the headline results, appeared
  nowhere in the file.
- It claimed pi "has no native Amazon Bedrock mode, so it cannot run
  --provider bedrock". pi bundles the bedrock-runtime client, the harness has a
  PI_PROVIDER_BEDROCK constant, and two committed scripts do exactly that.
- `--skill` was never documented as an input, while the text described `/swe2`
  throughout and the default is `swe3`.
- Every artifact path was missing two directory levels: the layout is
  <model>/<harness>/<skill>/<scope>/<task>, so the judge and summarizer commands
  as written pointed at a tree the harness never writes to.
- The default dataset was the 5-task v1 set rather than the 21-task v2 set the
  published results come from.
- Added the repository-reachability pre-flight and `api_key_env` from the
  previous commit, and notes on the context_window requirement behind a custom
  base URL.

The credential guidance now states that `credential.helper store` writes the
token in plaintext, offers the environment-only and deploy-key alternatives, and
says never to reach for GIT_SSL_NO_VERIFY when the fix is to trust the CA.

Registered in AGENTS.md, README.md, docs/repository-structure.md and
docs/benchmark-your-own-repo.md. All relative links verified to resolve.
@aarora79 aarora79 closed this Sep 29, 2026
@aarora79 aarora79 reopened this Sep 29, 2026
@aarora79 aarora79 changed the title Add private-repo pre-flight and env-sourced gateway key to the harness Add private-repo pre-flight, env-sourced gateway key, and the onboard-benchmark skill Sep 29, 2026
Step 1 checked that the machine reaches the gateway and the code host, but not
that it can reach a package index at all, which is what installing anything
depends on. A locked-down host often has none of the three.

Names PyPI (both pypi.org and files.pythonhosted.org, since wheels come from the
second and allowlisting only the first fails partway through the install),
registry.npmjs.org (the agent and the judge are npm packages), and the installer
and OS sources. Covers the internal-mirror settings for uv and npm, and the
NO_PROXY case where the gateway call gets routed to the proxy after everything
installs cleanly. Four matching rows added to the failure-mode table.
People assume the first step needs dedicated infrastructure. Anywhere Python
runs and dependencies can be installed works: an EC2 instance, a build or CI
runner, or a development workstation or laptop. No GPU, since the models are
hosted elsewhere.
Main added codex as a fifth coding agent (#162, #184), which needed reconciling
beyond the textual conflicts.

Textual conflicts, all in .claude/skills/benchmark/SKILL.md:

- The agent input. Kept this branch's table, added a `codex` row, and kept
  main's Responses-API tool-parser warning, which is hard-won and still true.
  Did NOT keep main's claim that pi cannot run --provider bedrock: pi has a
  PI_PROVIDER_BEDROCK constant and two committed scripts that do exactly that,
  which is one of the drift fixes this branch makes.
- The CLI pre-flight. Kept this branch's structure, added `codex exec --json`
  to the list of agent binaries.
- The orchestrator command line. Took both additions: codex in the agent list
  and the --skill flag.

One semantic conflict git could not see. Main's _build_codex_env sets
OPENAI_API_KEY from config.api_key directly, which bypasses this branch's
api_key_env and would send the "local" placeholder to a gateway that needs a
real credential. Routed it through resolved_api_key(), so all six call sites
that send the key now honour the environment variable.

Also updated the onboard-benchmark skill, which said codex was only the judge:
the harness-selection table now has a Responses-API row for it, and both that
step and the judge step note that running --agent codex puts one tool on both
sides of the run, so the judge model must differ from the model under test.
@aarora79
aarora79 merged commit 6a8e637 into main Sep 29, 2026
5 checks passed
aarora79 added a commit to shekharprateek/agentic-coding-harness-benchmarks that referenced this pull request Sep 29, 2026
Main gained the private-repo pre-flight, the env-sourced gateway key and the
onboard-benchmark skill (aarora79#193) while this branch was open.

Textual conflicts, all three in .claude/skills/benchmark/SKILL.md, where this
branch and aarora79#193 both rewrote the agent documentation:

- The agent input. Took aarora79#193's table form and added a `strands` row, rather than
  keeping the older prose list. The row records what the orchestrator actually
  enforces: strands runs on either provider, and unlike every other agent it has
  no CLI binary. Dropped the stale claim that pi cannot run --provider bedrock,
  which aarora79#193 corrected.
- The CLI pre-flight. Took aarora79#193's structure and kept this branch's strands fact
  as a following paragraph: there is no binary to find on PATH, the SDK runs
  in-process, and the orchestrator installs the optional dependency group with
  `uv sync --group strands`.
- The orchestrator command line. Took both additions, strands in the agent list
  and aarora79#193's --skill flag.

One semantic conflict git could not see. `_build_strands_env` sets the Strands
API key from `config.api_key` directly, which bypasses the `api_key_env` support
aarora79#193 added and would send the "local" placeholder to a gateway that needs a real
credential. Routed through `resolved_api_key()`, matching the other six call
sites. Verified: with api_key_env set, the strands env now carries the token from
the environment rather than the placeholder.

Also added strands to the harness-selection table in the onboard-benchmark
skill, which aarora79#193 introduced and this branch had no way to know about, so the
skill does not silently omit an agent.
@aarora79 aarora79 mentioned this pull request Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant